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I. 



INTRODUCTION 



A. BACKGROUND 

Programs for the acquisition of weapon systems are con- 
stantly facing new challenges. This is the result of infla- 
tion, rising costs and pressures both from Congress and the 
public. Elmer B. Staats, Comptroller General of the United 
States in 1974, states [1]: 

"The rising concern of the Congress and the public about 
increasing costs of new defense weapons is certainly not 
news. The patience of the public and the Congress has 
been worn thin by repeated experiences with contractor 
and Defense Department cost forecasts for new weapons 
that are unrealistic, and the promises of performance 
miracles by new weapons that later proved unattainable." 

In addition, today's defense strategies and tactics, 

designed to meet the rapidly changing international situations, 

require weapon systems of increasing complexity which demand 

an increase in the performance requirements and environmental 

operating conditions. This requirement for complex weapon 

systems has created a technology explosion which threatens to 

doom systems under development to technical obsolescence 

before they become operational. These problems were cited in 

1972 by former Deputy Secretary of Defense, David Packard [2]s 

"Cost overruns were the most visible symptoms of the 
troubled new weapon development situation but there were 
other problems too. Most programs took far too long from 
original conception until weapons were delivered to the 
operational military forces. As a result, many weapons, 
particularly those involving electronics and other fast- 
moving technology, were out of date by the time they were 
available. " 

To compound the problem, even after exorbitantly high cost 
and unnecessarily long development time, many of the new 
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devices did not have the reliability that is needed for mili- 
tary use. 

With these various factors affecting the acquisition of 
weapon systems, how does one achieve a highly reliable weapon 
system? Management in the Department of Defense and contrac- 
tors have increased their efforts to improve the reliability 
of weapon systems. However, despite these efforts, the reli- 
ability of weapon systems continues to be disappointing. 
Studies throughout the Department of Defense have been and 
continue to be made in an effort to find ways to improve 
reliability. 

In May 19 72, the ."Mean Time Between Failure" Improvement 
Study Group, headed by RADM K. R. Wheeler, SC USN, conducted 
a major study for the Navy on reliability and maintainability 
improvement. The resulting report, better known as the 
"Wheeler Report" [3] concluded that those equipments found 
unreliable in the fleet simply did not have reliability 
designed in from the beginning. The report cited the follow- 
ing specific conditions existing in 1972 which were considered 
to be crucial factors influencing reliability in Navy acquisi- 
tion efforts. 

Sufficient time and dollars omitted from budgetary 
submissions 

Unenforcible reliability goals in contracts and no 
enforcible reliability requirements 

Poor or nonexistent reliability testing procedures 

Insufficient reporting systems to identify unreliable 
equipments in the fleet 

Pressure for attaining specified performance goals 
which lead to trade-offs that though not explicit, 
aggregate to lower reliability 
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Poor documentation of trade-offs between reliability 
and performance 

In 1973 the Air Force also conducted a study [4] in which 
they found 37 possible deficiencies that could affect reli- 
ability of the weapon systems. The following is a listing of 
those that have a relation to the acquisition of weapon 
systems : 

Contractor has no incentive to overdesign reliability 
in equipment 

No provisions made for reliability growth during 
development 

Changes in mission profiles of operational equipment 

Operational equipment used in environments for which 
they were not designed 

Contract reliability requirements changed to goals 
or not enforced 

As a result of these Governmental studies, new reliability 
policies were issued. One emphasis being that quantitative 
reliability requirements will be stated in "Request for Pro- 
posals" (RFP) . The policies further state that these require- 
ments will be minimum acceptable, realistic and achievable. 

The effects of these new policies are still questionable. In 
the article "Improving R & D Management Through Prototyping," 
David Packard states [5]: 

"A serious problem that troubles all of our recent major 
programs is reliability. Numerous directives, specifica- 
tions, and other requirements have been placed on all 
major development programs to attempt to improve the reli- 
ability of new weapons. Very little improvement, if any, 
has come from this effort and very large sums of money 
have been spent. 

"Reliability cannot be achieved by adhering to detailed 
specifications. Reliability cannot be achieved by formula 
or by analysis. Some of these may help to some extent but 
there is only one road to reliability. Build it, test it, 
and fix the things that go wrong. Repeat the process until 
the desired reliability is achieved. It is a feedback pro- 
cess and no other way." 
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B. PURPOSE 



Reliability has become an issue of major concern in the 
acquisition of new weapon systems and today's reliability 
policies do not provide the program manager with the needed 
visibility during the weapon system's development. The pur- 
pose of this paper is to propose a method by which the program 
manager can monitor and measure the reliability growth of the 
weapon system as it proceeds through its life cycle from 
initial design to the end of development. 
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II. WHY A RELIABILITY PROGRAM 



One of the major forces that demands a reliability program 
is the ever increasing cost of weapon systems. This is well 
recognized by the Department of Defense as evident by their 
emphasis on the "Design to Cost" concept of which reliability 
is an essential element. This is further amplified by 
CDR Hollister, USN, and R. Shorey of the Office of the Assist- 
ant Director (Electronics) [6]: 

"In design to a cost development, reliability and maintain- 
ability are especially important considerations. There is 
concern that a developer, motivated to provide the highest 
level of system performance possible at a target cost, may 
implicitly trade reliability and maintainability for greater 
performance. . . 

"Minimum reliability and maintainability requirements are 
established along with other minimum acceptable performance 
specifications. These are treated as major system perform- 
ance parameters during performance verification." 

In his book "Reliability Mathematics," Bertram L. Amstradter 
states [ 7 ] : 

"The importance of obtaining highly reliable systems and 
components has been recognzied in recent years. From a 
purely economic viewpoint, high reliability is desirable 
to reduce overall costs. The disturbing fact that the 
yearly cost of maintaining some military systems in an 
operable state has been as high as ten times the original 
cost of the equipment emphasizes this need. . . 

"The need for and importance of reliability have been 
reflected in the constantly increasing emphasis placed on 
reliability by both government and commercial industry." 

Other factors that demand reliability improvement are 
safety, schedule delays, inconvenience, and combat effective- 
ness in the fleet. Willis J. Willoughby, Deputy Chief of 
Naval Material for Reliability and Maintainability, states 
[ 8 ]: 
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"The Navy, like other Military Services, is faced with 
less-than-desired combat effectiveness levels and rising 
life cycle costs. This situation exists because reli- 
ability has not been properly specified and pursued; 
rather, the Navy has relied heavily on integrated logistic 
support. " 
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III. PRESENT RELIABILITY POLICIES 



A. POLICIES 

Various instructions that have been issued provide the 
best overview of the present policies governing reliability 
and the acquisition of weapon systems. 

1. SECNAVINST 5000.1 

SECNAVINST 5000.1, System Acquisition in the Depart- 
ment of the Navy, was issued in March 1972 and contains as 
Enclosure (1), DOD Directive 5000.1. This instruction now 
establishes all major policy and management principles for 
the acquisition of weapon systems. In spite of the fact that 
reliability is now a major issue in the acquisition of weapon 
systems, no reference to reliability can be found in the 40 
pages of this instruction. In Section II, Conduct of Program, 
reference is made to Enclosure (4) of the instruction which 
is a list of applicable documents. This list contains no more 
than 56 different instructions that have some bearing on the 
acquisition of weapon systems. Buried in this list is 
SECNAVINST 3900. 36A, which is the instruction governing Reli- 
ability and Maintainability. 

2. SECNAVINST 3900. 33A 

SECNAVINST 3900. 33A, Initiation of Engineering and 
Operation Systems Development, was issued in September 1968. 
This instruction established the Navy policies for the acqui- 
sition of weapon systems during the Conceptual Formulation 
and Contract Definition phases. The following statements 
pertaining to reliability were contained in this instruction: 
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System Trade-offs: Trade-offs will be used to obtain 

optimum balance between total cost, schedule, and oper- 
ational effectiveness (reliability) for the system. 

Conceptual Phase: Quantitative reliability and maintain- 

ability goals and demonstration concepts. 

Engineering Development: Quantitative reliability and 

maintainability specifications for the system and major 
subsystems and proposed test plans to demonstrate their 
achievement. 

This instruction has been superseded with the issuance of 
SECNAVINST 5000.1. 

3. OPNAVINST 5000. 4 2 A 

OPNAVINST 5000. 42A, Weapon Systems Selection and 
Planning, was issued in March 1976. This instruction estab- 
lishes the policies for identifying operational requirements 
(OR) and conducting management reviews during system acquisi- 
tion. The instruction makes only two references to reliability. 
One being that reliability is to be considered in establishing 
system performance goals. The second reference is the follow- 
ing statement: 

From the outset, planning will accord high priority to 
simplicity in design and toughness of management, including 
trade-offs and contractual provisions, to ensure a high 
degree of reliability and maintainability. 

4. SECNAVINST 3900. 36A 

SECNAVINST 3900. 36A, Reliability and Maintainability 
(R & M) for Naval Material, was issued in June 1970. This 
instruction establishes the R & M policies for the Navy and 
assigns responsibility for its achievement. With regards to 
the acquisition of weapon systems, the following is found in 
this instruction: 

Basic Concept: Reliability techniques are based on the fact 

that failure will occur and they focus attention on means 
to minimize the effect of failure. The required R & M is 
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justified in that it supports mission requirements. 

Therefore, it must be considered throughout the life 
cycle of the system. 

Policy: During the conceptual and development phases of 

new systems, each program will include a minimum acceptable 
R & M value based on operational need. A specified confi- 
dence level is normally required for the R & M demonstration. 

Responsibilities : 

a. The Chief of Naval Operations is responsible for;. 

Ensure that requirements documents include numerical R & M 
requirements and "Goals" or "Objectives" shall not be used 
in lieu of R & M requirements. 

Evaluate proposed decreases in R & M requirements for impact 
on operational characteristics where development is unable 
to economically achieve R & M minimum values. 

b. The Chief of Naval Material is responsible for; 

Ensure that quantitative R & M requirements are responded 
to in proposal for industry. 

Determine adequacy of each contractor's R & M program for 
achieving and demonstrating R & M requirements. 



B. REQUEST FOR PROPOSALS 

To obtain a first hand knowledge of how these policies 
are implemented, three Request for Proposals (RFP) were 
reviewed. 

1. Harpoon Weapon System [9] 

a. Reliability Standards listed 

(1) MIL-STD 721: Definition of Effectiveness Terms 

(2) MIL-STD 756: Reliability Prediction 

(3) MIL-STD 781: Reliability Tests 

(4) MIL-STD 785: Reliability Program Plan 

(5) MIL-STD 1304A: Reliability Reports 

(6) MIL-R-22732: Reliability Requirement for 

Shipboard and Ground Electrical 
Equipment 

b. Reliability Section 

(1) Definition: A definition for reliability was 

provided in the RFP and it was the one cited in MIL-STD 721. 
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(2) Reliability Requirements: The requirements 

stated in the RFP were of two types, a reliability probability 
and a Mean Time Between Failure (MTBF) . Numerical numbers 
were stated, however, in the listing below only the reliability 
probability (R) or the MTBF will be noted. 

Sub System Min Acceptable Reliability 

Reliability Req't Design Objective 



Missile 

Freeflight 


R 1 


R 2 


Standby 


MTBF 1 


MTBF 


Ready 


MTBF 1 


MTBF 


Storage 


R 1 


R 2 


Aircraft Command 
and Launch 


MTBF 1 


MTBF 


Shipboard Command 
and Launch 


MTBF 1 


MTBF 



The RFP further amplifies the requirements by providing the 
following : 



Missile Freeflight 
a freeflight reliability of R 2 
as a minimum acceptable value, 
operate under a degraded mode, 
missile profile from launch to 

Aircraft Equipment 
equipment for aircraft command 
(numerically specified) design 
specified) minimum acceptable 



Reliability: They shall have 

as a design objective and Rj 
The missile shall be able to 
Free- flight is defined as the 
target intercept. 

Reliability: Each new item of 

and launch shall have a MTBF 
goal and a MTBF (numerically 



Shipboard Equipment Reliability: Same type of 

statement as for the aircraft equipment. 



(3) Reliability Program Plan: With regards to 

a reliability program plan, the RFP states the following: 

The contractor shall provide and maintain a reli- 
ability program that is acceptable to the government. 

The contractor shall use design techniques 
described in paragraph 5.2.1 of MIL-STD 785A to the extent 
practicable during the design phase to assure that the reli- 
ability requirements will be met during the weapon systems 
development phase. 
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The contractor shall conduct a reliability analysis 
of the Harpoon Weapon System during the design phase in accord- 
ance with paragraph 5.2.2 and all subparagraphs of MIL-STD 785A. 

2. VFAX Airborne Weapon System [10] , 

a. Reliability Standards listed 

(1) MIL-STD 721: Definition of Effectiveness Terms 

(2) MIL-STD 756: Reliability Prediction 

(3) MIL-STD 785: Reliability Program Plan 

b. Reliability Section 

(1) Reliability Requirements: Quantitative 

requirements are stated for two phases of the weapon system. 

Mission reliability: The mission reliability, 

expressed as the probability that the Airplane Weapon System 
can perform all the mission functions successfully, shall equal 
or exceed O.XX as a goal. 

Refly reliability: The refly reliability, expressed 

as the probability that the Airplane Weapon System can be re- 
turned to full operating capability without corrective main- 
tenance between missions, shall equal or exceed O.XX based on 
a t-hour mission duration with O.XX as a goal. 

(2) Reliability Program Plan: The request for 

proposal states that a reliability program will be established 
in accordance with MIL-STD 785 and the reliability specifica- 
tion appendix of the request for proposal. The appendix pro- 
vides amplification of various sections of MIL-STD 785 and 

in some cases specific instructions as to how and what will 
be included in the program. 

A section of the reliability specification 
appendix states what testing is required demonstrating the 
reliability of the weapon system. The tests are classified 
in two categories, preproduction and quality conformance. 

The preproduction test is to determine if the 
Airplane Weapon System has met the quantative reliability 
requirements. Whereas, the quality conformance test is to 
determine if the Airplane Weapon System offered for acceptance 
has met the quantitative reliability requirements. 

3. AEGIS Weapon System [11] 

a. Reliability Standards listed 

(1) MIL-STD 721: Definition of Effectiveness Terms 

(2) MIL-STD 756: Reliability Prediction 

(3) MIL-STD 781: Reliability Tests 
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(4) MIL-STD 785: Reliability Program Plan 

(5) MIL-R 22732: Reliability Requirements for 

Shipboard and Ground Electrical 
Equipment 

b. Reliability Section 

(1) Definitions: The RFP provides definitions 

for System Effectiveness, Inherent Availability and Operational 
Availability. 

(2) Reliability Requirements: The requirements 

stated in the RFP are of two types, those being availability 
and reliability. 

Shipboard System Availability: The minimum accept- 

able shipboard system inherent availability shall be at least 
O.XXX over a t-month deployment period under the following 
conditions : 



(a) The system will be in the standby or 
operate mode of operation at least 75 percent of the time. 

(b) The MFAR radar is to be considered a 
primary search radar. 

(c) The system is considered available when; 
It is down for preventive maintenance, provided the capability 
to return to full operation within t-minutes is retained. 

Five out of six illuminators are available and remainder of 
the system segments are providing specified performance. 

Shipboard System Reliability: The MTBF for the 

system, including test sets and built-in monitoring devices, 
but excluding missiles shall be not less than t-hours . 

(3) Reliability Program Plan: The reliability 

program plan requirement has been divided into three phases . 

C D Phase A : The contractor shall submit as part 

of his proposal a Preliminary Reliability and Maintainability 
Plan identifying the work to be performed during Engineering 
Development (ED) and its relationships to and expected impact 
on the designs formulation process. 

CD Phase B : The contractor shall, during this 

phase, expand the plan into a comprehensive program for execu- 
tion during engineering development. This program plan shall 
comply with requirements of MIL-STD 785 and MIL-STD 470. The 
plan shall present detailed procedures by which reliability 
and maintainability will be measured, evaluated, and con- 
trolled throughout ED. 

ED Phase : The contractor shall develop a Produc- 

tion R & M Plan. This plan shall establish and describe 
those detailed procedures by which inherent R & M character- 
istics are monitored and retained during production. 
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The RFP further breaks down the reliability pro- 
gram to detailed task requirements as follows: 

Requirements Interpretation, Allocation and 
Feasibility Analysis 
Predictive Assessments 
Design Selection 
Design Analysis 

Parts and Materials Selection and Control 

Parts Qualification 

Reliability Quality Assurance 

R & M Data Reporting, Analysis and Feedback 

Design Qualification Tests 

Acceptance Tests 

(4) Policies: To obtain maximum availability 

without sacrificing any of the high performance requirements, 
the following policies have been set forth: 

Maximum utilization of proven components — the 
design trade-offs must favor proven reliability over potential 
savings in weight, space or costs of unproven components. 

Standardization--Standardization of components, 
printed circuit cards, modules and subassemblies must be 
maximized. 



Redundancy and Fault isolators — the system design 
must be optimized for maximum reliability and minimum down 
time through the use of redundancy, automatic systems monitor- 
ing and simplified fault isolators. 



C. MIL-STD 785A 

Title: Reliability Program for Systems and Equipment 

Development and Production. 

Purpose: To establish uniform criteria for a reliability 

program and provide guidelines for the preparation and imple- 
mentation of a reliability program. 

Outline: This standard provides guidelines as to what 

shall be included in a reliability program plan. Major head- 
ings that are to be in this program plan are: 

Reliability Management 
Reliability Design and Evaluation 
Reliability Testing and Demonstration 
Failure Data 
Production Reliability 
Status Reports 
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As one reviews these various sections of the MIL-STD, it 
is noted that the context is in generalities and no specific 
detailed guidance is given. In the Program Review section 
(5.1.4) and the Design Review sections (5.2.7), it states that 
progress will be reviewed at appropriate stages of development 
and production to evaluate achievement of the reliability 
requirements. However, no information is given as to how the 
progress will be monitored or to what criteria it is to be 
measured against, other than the numerical requirement (which 
is the number to be achieved at the end of development) . 

D. SUMMARY 

A review of the policies has shown that very little guid- 
ance is given in terms of reliability and the acquisition of 
a weapon system. Guidance can be summed up in the following 
statement: 

A minimum acceptable numerical requirement and a Reliability 
Program requirement will be stated in the RFP . 

The three RFP's that were reviewed showed that this is 
exactly what was stated. As to the completeness and detail 
of the requirements, this depends upon the program manager 
who is responsible for the context of the RFP, as can be seen 
by the variance in the RFP's. In each of the RFP's, three to 
six military standards are referenced, but only one (MIL-STD 
785A) is cited in the reliability section. In the VFAX and 
AEGIS RFP's, additional detail is given to supplement MIL- 
STD 785A. However, there is no information given in any of 
the three RFP’s as to how the reliability of the system will 
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be monitored or measured during its development. As noted in 
the review of MIL-STD 785A, which is cited in all the RFP ' s , 
no information is contained as to monitoring or measurement 
of reliability during development. 
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IV. MANAGEMENT AND RELIABILITY GROWTH 



A. INTRODUCTION 

'Reliability Prediction" can be defined as an estimate of 
the performance we can expect of the equipment on its origi- 
nally scheduled delivery date. 

It is essential in treating the subject of reliability to 
understand basic concepts relating to techniques for predic- 
tion and control of reliability. The following contributions 
from persons knowledgeable in the field are intended to pro- 
vide this understanding. 

"The formal process of Reliability Prediction is based on 
the exponential distribution of random, statiscally inde- 
pendent failures, occurring as a result of some inherent 
nonsystematic propensity of things to fail. The mathematics 
associated with this process requires that failure rates of 
each constituent item be known and constant with time. The 
resulting failure rate "prediction" is independent of time. 
This is a fatal flaw, because we observe achieved failure 
rate to be a function of time. That being the case, the 
prediction can only represent an ultimate limiting value. 
Such a value might be of some use if it could be depended 
upon, but it cannot, because performance is exceeding pre- 
dicted values every day... 

"The numbers game has outlived its usefulness to the point 
of severely interfering with our ability to do the real 
job. In today's explosive development of new parts, and 
equipment of astonishing capability — where successive models 
are each a new generation of development — classical reli- 
ability prediction is nearly useless: The numbers it pro- 

duces are rarely relevant to the solution of any of our 
problems." (Ernest Codier, General Electric Co.) [12] 

"Many popular techniques for the analysis of reliability 
consider the problem only at a single point in time. Such 
techniques certainly yield value information. However, a 
complete treatment of system reliability requires careful 
consideration of the time variations introduced by design 
changes or modifications in maintenance practices." 

(J. T. Duane, General Electric Co.) [13] 
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"Experience has shown that reliability values reported from 
field service are generally lower — often much lower--than 
those predicted by design analysis or demonstrated by labor- 
atory testing. This has led to optimistic projections of 
operational effectiveness and ownership cost. It can also 
lead managers down a "primrose path" during development, 
since predictions and demonstrations indicate that reli- 
ability requirements are being met." (Robert Parker, Deputy 
Director Defense Research and Engineering) [14] 

"...Despite special management efforts and major techno- 
logical improvements. Air Force avionics equipment still 
does not demonstrate high reliability. This shortcoming 
is compounded by the fact that the reliability predictions 
which are formulated during equipment design, refined during 
development, and tested during formal laboratory demonstra- 
tions, infer equipment reliability that is often signifi- 
cantly greater than what is actually achieved when the 
equipment is operationally employed." (Lt Gen Robert Marsh, 
USAF, Vice Commander, Air Force Systems Command) [15] 

As noted by the various comments, the conventional reli- 
ability analysis techniques, in particular "reliability pre- 
diction," are not providing the desired reliability in weapon 
systems. Reliability prediction provides a useful data point. 
However, it is being used in dealing with time and design 
changes, but it does not provide the manager with a useful 
tool to monitor the reliability growth of the weapon system 
during its development. In the implementation of the present 
reliability policies (Section III) , the manager is in the dark 
as to the reliability of the weapon system until development 
is completed. In order to insure that the weapon system reli- 
ability will meet the contractual requirement, the program 
manager must have some type of yardstick by which he can 
visually measure the reliability growth of the weapon system 
during development. This yardstick must also provide inform- 
ation as to when design changes are required, and when the 
system has met its contractual requirement. Barlow states: 
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"It is common practice, during the development of a system 
to make engineering changes as the program develops . These 
changes are generally made in order to correct design defici- 
encies and, thereby, to increase reliability. This elimina- 
tion of design weaknesses is what we mean by reliability 
growth." [17] 

To summarize the above, reliability growth is that growth 
observed in reliability as the weapon system progresses along 
its life cycle. A visual representation is presented in 
Figure 1. 

B. THE YARDSTICK 

Present policy requires that a reliability program be 
established for a new weapon system in accordance with MIL- 
STD 785A. This military standard provides for a sound reli- 
ability program for the attainment of the reliability require- 
ment. However, it does not provide the means to monitor or 
measure the reliability growth of the weapon system. It is 
proposed that a "Reliability Growth Curve Prediction" be con- 
tractually imposed as part of the Reliability Program Plan to 
provide the means to monitor and measure reliability. 

C. RELIABILITY GROWTH CURVE 

1. What is It ? Amstradter states: 

"...reliability growth is, in reality, a discrete function 
which occurs, hopefully, each time a design change is 
made. [16] 

2 . Is It New ? 

Reliability growth is not new. Much has been written 
about it, as can be noted in Appendix A, Bibliography on 
Reliability Growth. It has only been in recent years that it 
has come into its own and has been recognized as a useful 
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TYPICAL RELIABILITY GROWTH CURVE 
FIGURE 1 



management tool. Although the major emphasis has been only 
to monitor reliability growth during development testing, 
more thought is being given to use it as a forecasting 
method. 

A reliability growth model is an analytic tool useful 
in determining the time required to develop an acceptable 
product. The model monitors the progress of the develop- 
ment program and can be used to forecast the time required 
to achieve the reliability goal if the program were to be 
modified. [18] 

This technique should be useful to the reliability 
engineer in the monitoring of development progress to 
insure that reliability specification requirements are 
achieved. A further desirable characteristic would be 
for techniques to provide for continuing extrapolation 
from current status to some future result... This tech- 
nique is known as reliability growth curve methodology. [19] 

3 . Two Types of Growth Curves 

Reliability growth curves can be classified into two 
categories; Assessment and Apportionment [20]. 

a. Assessment--When reliability-growth equations and 
curves are prepared after a program is well into the develop- 
ment phase and thereby utilize actual test data to arrive at 
appropriate numerics, growth analysis becomes primarily an 
assessment process. 

b. Apportionment — If a growth curve is prepared to 
depict expected levels of reliability achievement and to pro- 
vide a continuum of time-oriented goals, it can be classed as 
an apportionment procedure. 

4. Models 



There are several reliability growth models, each of 
which is valid in its own right. The purpose of this paper 
is not to present the various models and how to apply them. 
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but to show that reliability growth curves have a place in 
management. However, to acquaint the reader with reliability 
growth models, Appendix B provides a review. 

5 . The Purpose of Reliability Growth Models 

The use of reliability growth curves has several 
potential purposes. They include: 

a. Monitoring the reliability progress of the system 
as it proceeds through development. 

b. If a reliability growth curve was predicted, it 
provides a means to measure the actual growth of the system. 

c. It provides for forecasting of reliability in the 
short term. 

d. Provides the means to measure the effectiveness 
of design changes. 

e. Aids in defining design problems and what their 
impact on reliability will be. 

f. Aids in the allocation and reallocation of resources 
to achieve requirements on schedule and within constraints. 

g. Aids in determining requirements for future 
procurements . 

h. Provides a useful communication vehicle for high 
level management. 

i. Aids in planning the development program and to 
control its progress as the design matures. 

6 . Is It Applicable ? 

The best method to show the applicability of reli- 
ability growth curves is to cite several examples. The 
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examples will be divided into the two categories of reliability 
growth modeling; assessment and apportionment, 
a. Assessment Examples 

(1) F-15A Aircraft [21]. The reliability program 
for the F15A did not call for reliability growth modeling. 

The reliability requirements were typical of today's policies, 
i.e., MTBF and a reliability program plan. As part of the 
reliability program, data were collected, analyzed and plotted. 
This plotted data (Figure 2) shows the reliability growth of 
the aircraft during development testing. 

It should be noted that the growth curve is 
not continuous from Category I (contractor evaluation) to 
Category II (Air Force evaluation) . The major contributing 
factor for this discontinuity is the change in operational 
environment. This is one of the considerations that must be 
recognized in using reliability growth modeling. 

(2) TOW Weapon System [22]. The TOW weapon sys- 
tem is a heavy assault weapon for the U. S. Army. The reli- 
ability requirements for this program were based on present 
day policies. However, a strong emphasis was stressed in the 
"find and fix" and "overstress" test approach to reliability. 
Although reliability growth modeling was not a contractual 
requirement, a plot of reliability growth was maintained 
throughout the program (Figure 3) . 

(3) F-111A Aircraft [23]. The reliability pro- 
gram for the U. S. Air Force F-111A aircraft was conducted 
under present day policies . To assure that the requirements 
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were obtained, a rigorous reliability program was imposed on 
the contractor. Reliability growth prediction was not part 
of this program; however, data were plotted to show the reli- 
ability growth of the aircraft during its development (Figure 
4) . 

b. Apportionment Examples 

(1) C-141 Aircraft [24]. The U. S. Air Force 

reliability requirement for the C-141 was as follows: The 

contractor shall plan and conduct a comprehensive reliability 
program in accordance with the requirements of Weapon Systems 
Specification MIL-R-26674, Reliability Requirements for Wea- 
pon Systems. 

Although the Air Force did not specify a 
numerical reliability requirement, the contractor established 
a design goal and performed a prediction analysis to show 
projected growth to this goal. As the program progressed, 
predictions were continually updated which resulted in adjust- 
ment to the growth curve (Figure 5) . As the program progressed 
through development, observed growth was plotted against the 
predicted growth curve (Figure 6) . This visual representa- 
tion served as a basis for program reviews and direction for 
the contractor and the Air Force. 

(2) SATCOM Terminals [25]. Reliability growth 
prediction was not imposed as a contractual requirement, but 
the Army used the Duane and Weibull reliability growth models 
to make reliability management decisions. These growth 
models helped to plan the growth of reliability in the develop- 
ment program' and to control its progress as the design matured. 
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The authors of this article summed up the use of this technique 
in the following statement: 

"The Duane and Weibull growth curves were used to deter- 
mine requirements for future buys, provide management 
visibility for reliability and make decisions regarding 
test termination." 

(3) SM-2 Missile [26]. To meet the contractual 
MTBF requirement, the contractor established a three-phase 
reliability program. One phase of this program is "Reliabil- 
ity Growth Monitoring." The procedure that the contractor 
set up to accomplish this phase was to plot a predicted growth 
curve that the system was expected to take during its develop- 
ment. The observed data will be plotted against this predic- 
tion to provide the impetus for additional action if it appears 
that the MTBF requirement might not be met. This approach is 
being taken by the contractor mainly due to another program 
in which the MTBF growth was plotted and found to be a useful 
management tool. In a conversation with J. C. Bear [27], he 
commented that General Dynamics is finding this technique very 
useful and feels that the potential is there for a very power- 
ful management tool . 

D. THE MANAGEMENT TOOL 

It has been shown that a potential management tool exists 
in the use of Reliability Growth Curves. Although the major- 
ity of the interest to date has been the assessment of reli- 
ability growth, interest is beginning to stir in its use as 
an estimator of reliability growth of a weapon system. This 
is evident by contractual requirements now being imposed by 
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the U. S. Army on new weapon system developments, such as their 
Stinger program [28], 

Present day policies do not provide the program manager 
visibility as to the reliability growth of the system. The 
manager does not become aware that the system, in its present 
design, will not meet the requirements until it is well into 
demonstration testing. With a predicted reliability growth 
curve, the program manager has the means by which to measure 
actual growth and to aid in decision making as to design 
changes and testing of the system. 

E . IMPLEMENTATION 
1. Requirement 

The requirement for a "Reliability Growth Curve 
Prediction" should be part of the Reliability Program Plan 
requirement of the RFP. The requirement could take one of 
two forms : 

a. The Navy with their knowledge of the new weapon 
system design and development time constraints, could submit 
the desired Reliability Growth Curve to which the contractor 
could respond as they presently do with the MTBF requirement. 

b. The Navy could require that the contractor submit 
a "Reliability Growth Curve" as part of their reliability pro- 
gram plan. 

In either case, because of its sensitivity depending upon the 
model used and the factors involved in predicting the curve, 
the final Reliability Growth Curve Prediction will be a nego- 
tiated item between the Navy and the contractor. 
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2. Pit-Falls 



In that Reliability Growth Curve Prediction would be 
a new management technique, pit-falls should be expected in 
the early stages. All factors that affect the prediction and 
the exact sensitivity to the different models can only be 
learned with use. The following is a listing of several fac- 
tors that must be considered in selecting the appropriate 
model and the use of Reliability Growth Curve Prediction. 

a. Complexity of the weapon system in that it will 
have a definite bearing on expected failure rate. 

b. Schedule time alloted for development of the sys- 
tem. (Short time, you would expect to see a rapid 
growth; long time, a more gradual growth.) 

c. The various environments that the system is sub- 
jected to during development. (Laboratory, con- 
tractor field testing, customer testing, etc.) 

d. Definition of failure: To avoid discontinuities 

in the growth curve, failure definition must remain 
constant [27]. 

e. Number of expected failures: This would have a 

bearing in selecting the model that would provide 
the best growth fit for the system. 
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V. CONCLUSIONS 



The program manager, today, has no means available to him 
to provide visibility with regards to reliability growth. 
Reliability prediction provides the manager with the potential 
maximum value at that point in time when the prediction is 
made. Reliability demonstration provides the manager with the 
actual value at the time when the test was conducted. Neither 
of these two methods provides the manager with the answer to: 
"Is the weapon system going to meet the reliability 
requirement? " 

The "Reliability Growth Curve Prediction" would provide 
the manager with insight as to how the weapon system should 
meet the reliability requirement. And when actual data are 
plotted and compared to the predicted curve, it would provide 
the manager insight to the question: "Is the weapon system 

going to meet the reliability requirement?" 
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APPENDIX B 

A REVIEW OF SOME RELIABILITY GROWTH MODELS 

This appendix describes a number of reliability growth models which 
are currently available. Each model is briefly described including the 
basic assumptions that were made in deriving the models. Technical 
references are given for each of these models where a more complete 
discussion of the model may be found. 

Model 1. Lloyd and Lipow (see Appendix B, Reference 28) introduced a 
reliability growth model for a system which has only one failure mode. 
For each trial it is assumed that the probability is a constant that 
the system will fail if the failure mode has not been previously 
eliminated. If the system does not fail, no corrected action is per- 
formed before the next trial. If the system fails, then an attempt is 
made to remove the failure mode from the system. The probability of 
successfully removing the failure mode is also assumed to be a constant, 
for each attempt. They show that the system reliability, R n> on the 
n-th trial is 



where A and C are parameters. 

Model 2. Another reliability growth model was considered by Lloyd and 
Lipow (see Appendix B, Reference 28) where the development program is 
conducted in K stages and on the i-th stage a certain number of systems 
are tested. The reliability growth function considered was 

R. = R -a/i, 

l 00 

where R i is the system reliability during the i-th stage, R^ is the 
ultimate reliability as i-»« and a>0 is a parameter. Maximum likelihood 
and least squares estimates of R^ and a are given by Lloyd and Lipow 

* Ref: Crow, Larry H. Reliability Growth Modeling , US Army Material 

System Analysis Activity, Aberdeen Proving Ground, MD, Tech Rot 55, 1972. 
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along with a lower confidence limit for R^,. 



Model 3. Weiss (see Appendix B, Reference 41) considered a reliability 
growth model where the mean time to failure of a system with exponential 
life distribution is increased by removing the observed failure modes. 

In particular, he showed that when certain conditions hold, the increase 
of the mean time to failure is approximately at a constant percent per 
trial. That is, if 0(i) is the mean time to failure of the system at 
trial i then 9(i) may be approximated under certain conditions by 

6(i) = Ae^, 

where A and C are parameters. Note that 

Q(i+1) = e C 0 (i) . 

The maximum likelihood estimates of A and C are given by Weiss. 

Model 4. Wolman (see Appendix B, Reference 42) considered a situation 
where the system failures are classified according to two types. The 
first type is termed "inherent cause" and the second type is termed 
"assignable cause". Inherent cause failures reflect the state-of-the 
art and may occur on any trial while assignable cause failures may be 
eliminated by corrective action, never to appear again. Wolman assumed 
that the number of original assignable cause failures is known and that 
whenever one of these modes contribute a failure, the mode is removed 
permanently from the system. Wolman uses a Markov-chain approach to 
derive the reliability of the system at the n-th trial when the failure 
probabilities are known. 

Model 5 . Barlow and Scheuer (see Appendix B, Reference 3) considered 
a nonparametTic model for estimating the reliability of a system during 
a development program. They assumed that the design and engineering 
changes do not decrease the system's reliability, but, unlike some 
other models, they do not fit a prescribed functional form to the 
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reliability growth. Their model is similar to Wolnan's in that each 
failure must be classified either as inherent or assignable cause. 

It is further assumed that the development program is conducted in 
K stages, with similar systems being tested within each stage. For 
each stage, the number of inherent failures, the number of assignable 
cause failures and the number of successes are recorded. In addition, 
they assumed that the probability of an inherent failure, a Q , remains 
the same throughout the development program and that the probability of 
an assignable cause failure, q^, in the i-th stage does not increase 
from stage to stage of the development program. The authors obtained 
the maximum likelihood estimates of q Q and of the q.. T s subject to the 
condition that they be nonincreasing. A conservative lower confidence 
bound for the reliability of the system in its final configuration was, 
also, given. 

Model 6 . Virene (see Appendix B, Reference 38) considered the suitability 
of the Gompertz equation 



R = ab C , 

0 < b < 1, 0 < c < 1, for reliability growth modeling. In this equation 
a is the upper limit approached by the reliability R as the development 
time t— The parameters a, b and c are unknown. Virene gave estimates 
of these parameters and demonstrated by examples the application of this 
model. 

Model 7 . Duane (see Appendix B, Reference 17) considered a deterministic 
approach to reliability growth modeling. He analyzed data available 
for several systems developed by General Electric in an effort to 
determine if any systematic changes in reliability improvement occurred 
during the development programs for these systems. His analysis revealed 
that for these systems, the cumulative failure rate versus cumulative 
operating hours fell close to a straight line when plotted on log-log 
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paper. The cumulative failure rate appeared to decrease at approximately 
the -0.4 or -0.5 power of cumulative operating hours. 

The types of systems investigated were of the complex electro- 
mechanical and mechanical nature. Duane concluded that a line with a 
slope of -0.5 representing cumulative failure rate as a function of 
cumulative operating hours on log-log paper would probably be suitable 
for reflecting reliability growth for similar type systems developed at 
General Electric. 

Mathematically, Duane's failure rate equation may be expressed by 

\ (T) = KT" a , 

K > 0, 0 <_ a <_ 1, where A(T) is the cumulative failure rate of the 
system at operating time T, and K and a are parameters. It 
follows then that 



*(T) = MU 
T 



where E(T) is the expected number of failures the system will 
experience during T units of operation. This yields 



E(T) = KT 1 " 0 . 



Furthermore, the instantaneous failure rate at T is given by 



6(T) = (l-a)KT" a . 



For a system with a constant failure rate the mean time between failur e 
(MTBF) of the system at operating time T^ is / , 



M(T) = [OCT)] -1 = [(l-cOKrV. 



That is, the change in system MTBF during development is proportional 



to T a . 



46 



With this notation a =* 0.5 closely represented the types of systems 
considered by Duane. 

Model 8. Pollock (see Appendix B, Reference 32) considered a Bayesian 
reliability growth model for a system undergoing development. The 
parameters of the model are assumed to be random variables with 
appropriate prior distribution functions. Using his results, one may 
project the system reliability to any time after the start of the 
development program without data and, also, estimate the system 
reliability after data have been observed. He further gave precision 
statements regarding the projection and estimation. 

Model 9 . Barlow, Proschan and Scheuer (see Appendix B, Reference 2) 
considered a reliability growth model which assumes that a system is 
being modified at successive stages of development. At stage i the 
system reliability (probability of success) is p^. The model of 
reliability growth under which one obtains the maximum likelihood 
estimates of p^ ,p 2 , . , . ,p^ assumes that 



That is, it is required that the system reliability be not degraded 
from stage to stage of development. No particular mathematical form 
of growth is imposed on the reliability. In order to obtain a con- 
servative lower confidence bound on p^, it suffices to require only 
that 



p„ > max p. 
i<K 



That is, it is only necessary that the reliability in the latest stage 
of development be at least as high as that achieved earlier in the 
development program. 

Data consist of x. successes in n. trials in sta^e i i=l K 
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A variation of this model is treated in Barlow and Scheuer (see 
Model 5). In that model two types of failure, inherent and assignable 
cause, are distinguished. 

Model 10 , Another reliability growth model considered by Barlow, 
Proschan and Scheuer (see Appendix C, Reference 2) assumed that at 
stage i of development the distribution of system life length is F^. 
The model of reliability growth under which the maximum likelihood 
estimates of F^ (t) ,F 2 (t) , . . . ,Fj,(t) are obtained, writing 

F.(t) = l-F.(t) 

is 

Fj(t) £ F 2 (t) < ... £ F K (t) 

for a fixed t >_ 0. In order to obtain a conservative upper confidence 
curve on F^ft) and thereby, a conservative lower confidence curve on 
Pj,(t) for all non-negative values on t, it suffices only to require 
that 



F K (t) >_ max F. (t) 
i<K 1 



for all t >_ 0. That is, the probability of system survival beyond any 
time t in the latest stage of development is at least as high as that 
achieved earlier in the development program. 



Data consist of independent life length observations 



X il — 
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K. 



Model 11 . Barlow, Proschan and Scheuer (see Appendix C, Reference 2), 
also, considered a reliability growth model which assumes that the 
system life at the i-th stage of development has increasing failure rate 
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Because of improvement from stage to stage 



r ! (*) 1 r 2 (t) - * * ‘ - r K^ t ' ) 

for t >_ 0, where r^(t) is the failure rate at time t at the i-th stage 
of development. That is, for each t >_ 0, the probability of system 
failure in the interval (t, t+dt) , given survival till time t, does not 
increase from stage to stage of the development program. 

Given life- length observations, X. , ,X. 0 , . . . ,X. , the maximum 
° ’ ll u in 

likelihood estimates of r^ (t) ,r 2 (t) , . . . ,r^(t) are obtained. 
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